Rapid Server Rapid Server DeploymentDeployment

Presented by Matthew BulatPresented by Matthew Bulat

ObjectivesObjectives

New server considerationsNew server considerations Redundancy optionsRedundancy options Site preparationSite preparation Deployment documentationDeployment documentation Deployment toolsDeployment tools System Testing System Testing

BackgroundBackground

Recent Server Deployment Project Recent Server Deployment Project 42 old NT4 servers being replaced 42 old NT4 servers being replaced

with Windows 2003with Windows 2003 Genesys Call Centre Software Genesys Call Centre Software

upgradedupgraded 16 Sites around Australia16 Sites around Australia Most deployed in February 2006Most deployed in February 2006 Team group of 4 personsTeam group of 4 persons No Active DirectoryNo Active Directory

My RoleMy Role

Hardware ChoiceHardware Choice Site ScopeSite Scope Standard Operating SystemStandard Operating System Implementation DocumentationImplementation Documentation Design deployment toolsDesign deployment tools Deployments in Sydney, Brisbane, Deployments in Sydney, Brisbane,

Penrith and CanberraPenrith and Canberra

Server ConsiderationsServer Considerations Software NeedsSoftware Needs ProcessingProcessing StorageStorage MemoryMemory Network ConnectionsNetwork Connections Redundancy OptionsRedundancy Options Future growth Future growth

potentialpotential Physical EnvironmentPhysical Environment

Redundancy OptionsRedundancy Options Dual ProcessorDual Processor Dual Power SupplyDual Power Supply Multiple Redundant Multiple Redundant

Cooling fansCooling fans RAID hard drivesRAID hard drives Multiple memory Multiple memory

slotsslots Multiple network Multiple network

socketssockets Same model servers Same model servers

to have whole server to have whole server backupbackup

Site EquipmentSite Equipment

Keyboard Video Keyboard Video Mouse SwitchMouse Switch

Network Switch Network Switch connectionsconnections

Power Power RequirementsRequirements

CoolingCooling Backup methodsBackup methods RackingRacking

Site ChecklistSite Checklist

Space for server/sSpace for server/s Rack ventilation for higher powered Rack ventilation for higher powered

serversservers Power points available (more for Power points available (more for

redundant power)redundant power) UPS capacity for new load for serversUPS capacity for new load for servers Keyboard Video Mouse cable lengthsKeyboard Video Mouse cable lengths Patch Panel / Network Switch spacePatch Panel / Network Switch space Network Cable LengthsNetwork Cable Lengths

Server ChoiceServer Choice

2 RU Server2 RU Server Dual 3.4 GHz XeonDual 3.4 GHz Xeon Mirrored 73 GB drivesMirrored 73 GB drives 4 x 512 Mb RAM4 x 512 Mb RAM 4 + 4 redundant cooling 4 + 4 redundant cooling

fansfans Redundant Power 600WRedundant Power 600W Dual Gigabit networkDual Gigabit network DVD burner on 1 server DVD burner on 1 server

per siteper site

Server LayoutServer Layout

Windows 2003 ServerWindows 2003 Server

Windows 2003 Standard allows 4 CPU Windows 2003 Standard allows 4 CPU 4GB Ram4GB Ram

Server roles added as requiredServer roles added as required Roles have high level of security by defaultRoles have high level of security by default DNS name resolution standardDNS name resolution standard Support IPv6 and IPv4 TCP/IP addressingSupport IPv6 and IPv4 TCP/IP addressing Distributed File System – Allow a logical Distributed File System – Allow a logical

file structure that can be replicated at file structure that can be replicated at many sitesmany sites

Windows 2003 continuedWindows 2003 continued

Remote Desktop has unlimited connectionsRemote Desktop has unlimited connections Windows 2003 Enterprise allows Cluster Windows 2003 Enterprise allows Cluster

setupssetups Internet Information Server 6 more resilient Internet Information Server 6 more resilient

to crashing with multiple pool processesto crashing with multiple pool processes Pools can have set properties and are Pools can have set properties and are

recycledrecycled IIS6 has read only static web pages by IIS6 has read only static web pages by

default initially default initially

.NET framework.NET framework Installed on Windows 2003 / XP or can be Installed on Windows 2003 / XP or can be

downloaded on Windows 98+downloaded on Windows 98+ Provides standard objects for programming with Provides standard objects for programming with

the operating system. E.g. Win32, IO, Threading, the operating system. E.g. Win32, IO, Threading, System Management, Drawing, Forms, Data, System Management, Drawing, Forms, Data, Web Services, Messaging and Directory Services.Web Services, Messaging and Directory Services.

Dynamic Link Library use is reduced using .NETDynamic Link Library use is reduced using .NET Finished programs are much smaller as a lot of Finished programs are much smaller as a lot of

the work is achieved by referencing .NETthe work is achieved by referencing .NET .NET framework can be used by multiple .NET framework can be used by multiple

languageslanguages

Standard Operating Standard Operating EnvironmentEnvironment

List all software usedList all software used Design of the Operating System Design of the Operating System

Configuration e.g.. Unattend.sifConfiguration e.g.. Unattend.sif List all media requiredList all media required Anti Virus selectedAnti Virus selected Security PolicySecurity Policy Server Management tools e.g. MOM, Server Management tools e.g. MOM,

ESMPROESMPRO

Microsoft Baseline Security Microsoft Baseline Security AnalyserAnalyser

Free download (complete process in 1 Free download (complete process in 1 day)day)

Make sure patches are up to dateMake sure patches are up to date Load on laptopLoad on laptop Crossover to target computerCrossover to target computer State server name to testState server name to test Wait for reportWait for report Review deficiencies and alter security Review deficiencies and alter security

policypolicy

Set Server Unattend.vbsSet Server Unattend.vbs

INPUTSINPUTS Input Server NameInput Server Name Unattend sourceUnattend source Unattend Unattend

variables, Server variables, Server Name, IP Address, Name, IP Address, Subnet Mask, Subnet Mask, Gateway, DNS, Gateway, DNS, Time Zone, Time Zone, License KeyLicense Key

OUTPUTSOUTPUTS Winnt.sif Unattend Winnt.sif Unattend

OutputOutput Set_Network.batSet_Network.bat Set_DNS.batSet_DNS.bat Who_Am_I.txtWho_Am_I.txt

Configuration FloppyConfiguration Floppy

Set Server Unattend.vbsSet Server Unattend.vbs Local Security PolicyLocal Security Policy Post Operating System Batch file for Post Operating System Batch file for

Drive letters, directories, shares, Drive letters, directories, shares, Accounts, Security policy, time sync Accounts, Security policy, time sync sourcesource

SCSI / RAID driversSCSI / RAID drivers Server Check toolsServer Check tools Host file Host file

Server Checks.vbsServer Checks.vbs Uses Windows Management InstrumentationUses Windows Management Instrumentation Configuration file used to display select Configuration file used to display select

HTML outputHTML output Confirms Server name, hardware Confirms Server name, hardware

specifications, hardware operational, display specifications, hardware operational, display settings, individual service status, user settings, individual service status, user accounts etc.accounts etc.

Result in form Parameter, expected value, Result in form Parameter, expected value, returned value (green = match, red = not returned value (green = match, red = not match)match)

Win32_DiskDrive,Status,OK,=, Win32_DiskDrive,Status,OK,=, OKOK

Media ListMedia List

Windows 2003 CDWindows 2003 CD NEC Express Builder NEC Express Builder

CDCD Software CDSoftware CD SQL CDSQL CD Configuration FloppyConfiguration Floppy Management Virtual Management Virtual

MachineMachine SQL Virtual MachineSQL Virtual Machine

NEC Express Builder CDNEC Express Builder CD It has all drivers for serverIt has all drivers for server You can create a Management Utilities You can create a Management Utilities

partitionpartition You can run diagnostics from the CDYou can run diagnostics from the CD Has ESMPRO server monitoring and Has ESMPRO server monitoring and

control software including Wake on LAN, control software including Wake on LAN, reboot ability, utilization monitoring, reboot ability, utilization monitoring, temperatures, voltages, fan speeds, CPU temperatures, voltages, fan speeds, CPU load, drive usageload, drive usage

Has Diana Scope which can read a standard Has Diana Scope which can read a standard hardware monitoring card within the server hardware monitoring card within the server including log files held in flash memoryincluding log files held in flash memory

Management Virtual Management Virtual MachineMachine

Imitate central management serverImitate central management server Deploy Windows Server Update Deploy Windows Server Update

Service connection with recent Service connection with recent Microsoft Service Packs, Patches / Microsoft Service Packs, Patches / Security updatesSecurity updates

Deploy Anti Virus with recent Deploy Anti Virus with recent definitionsdefinitions

Use ESMPRO to test server hardware Use ESMPRO to test server hardware earlyearly

SQL Server Virtual SQL Server Virtual MachineMachine

Imitate central SQL Server structureImitate central SQL Server structure This allows software installation with This allows software installation with

a known SQL server structure a known SQL server structure presentpresent

Allow running of DTS packages to Allow running of DTS packages to set up databasesset up databases

Allows the setting up and testing of Allows the setting up and testing of authentication structuresauthentication structures

Windows Server Update Windows Server Update ServiceService

Provides a single point for updates on your Provides a single point for updates on your networknetwork

Free software, needs IIS setup, Web InterfaceFree software, needs IIS setup, Web Interface Update information to WSUS, you choose Update information to WSUS, you choose

updates to download and computer groupsupdates to download and computer groups 8 GB of updates available 8 GB of updates available Script to configure PC to use of WSUS Script to configure PC to use of WSUS Uses Background Intelligence Transfer ServiceUses Background Intelligence Transfer Service WSUS knows all updates deployed on each clientWSUS knows all updates deployed on each client Can export WSUS data and import it into a Can export WSUS data and import it into a

remote network WSUSremote network WSUS

Microsoft Operations Microsoft Operations Manager 2005Manager 2005

Group Management of computersGroup Management of computers Real time analysis of issues, performance and Real time analysis of issues, performance and

securitysecurity Health of network can be seen in single viewHealth of network can be seen in single view Alerts can be drilled down to computer group, Alerts can be drilled down to computer group,

computer, processcomputer, process Automatic / manual responses can be done Automatic / manual responses can be done

locally or centrallylocally or centrally Computer types can have their own rule setComputer types can have their own rule set Standard rule sets available free or via 3Standard rule sets available free or via 3rdrd

party e.g. Ciscoparty e.g. Cisco

MOM continuedMOM continued Alerts can be logged in/out to personAlerts can be logged in/out to person Microsoft knowledge base available with Microsoft knowledge base available with

context of alertcontext of alert Company knowledge base built as alerts Company knowledge base built as alerts

handledhandled History of actions in similar alert availableHistory of actions in similar alert available Network connections constantly monitoredNetwork connections constantly monitored Test transactions can be used to monitor Test transactions can be used to monitor

performanceperformance WMI style computer discovery abilityWMI style computer discovery ability 100 standard reports for resource availability, 100 standard reports for resource availability,

capacity planning, alert response time, alert capacity planning, alert response time, alert type and frequencytype and frequency

MOM representationMOM representation

MOM RequirementsMOM Requirements

MOM on SQL ServerMOM on SQL Server Clients can report to up to 4 MOM serversClients can report to up to 4 MOM servers Suggest <= 10 Admin consoles per MOM Suggest <= 10 Admin consoles per MOM

(30 max)(30 max) Clients can be XP / 2000 / 2003Clients can be XP / 2000 / 2003 100 Clients generate approx. 250 MB / day100 Clients generate approx. 250 MB / day Maximum 2000 clients per MOMMaximum 2000 clients per MOM 2 minutes response time to Admin Console2 minutes response time to Admin Console Approximately 2% Proc. client load, Approximately 2% Proc. client load,

100Mb space, 10 Mb RAM100Mb space, 10 Mb RAM

MOM Admin ConsoleMOM Admin Console

Computer Telephony Computer Telephony IntegrationIntegration

Genesys SoftwareGenesys Software Translator – matches PABX phone activity and Translator – matches PABX phone activity and

PC activity to stay in sync. Common API PC activity to stay in sync. Common API independent of PABX model. Genesys allows a independent of PABX model. Genesys allows a whole extra layer of customization. E.g. Route whole extra layer of customization. E.g. Route by skill, client representative, client team, by skill, client representative, client team, client region.client region.

SQL Server – Defines rules for routing calls, SQL Server – Defines rules for routing calls, caches Solution Configuration, keeps call logs caches Solution Configuration, keeps call logs and statisticsand statistics

Management – Configure PABX, configure Management – Configure PABX, configure Genesys, monitor and control GenesysGenesys, monitor and control Genesys

IVR – Receives call, collects key data to pass IVR – Receives call, collects key data to pass on to Genesys for call handling purposeson to Genesys for call handling purposes

Genesys MonitoringGenesys Monitoring Each server runs a Local Control AgentEach server runs a Local Control Agent LCA can start, stop and monitor LCA can start, stop and monitor

applications under its control.applications under its control. LCA reports to Solution Control ServerLCA reports to Solution Control Server You use Solution Control Interface to You use Solution Control Interface to

look at SCS for current status and to look at SCS for current status and to control the solution.control the solution.

Network connections are monitored Network connections are monitored constantlyconstantly

Alarms can be sent in various formsAlarms can be sent in various forms

Server Redundancy Server Redundancy StylesStyles

Have applications loaded on more than 1 server Have applications loaded on more than 1 server with 1 active and the others disabledwith 1 active and the others disabled

Move mission critical hard drives from one server Move mission critical hard drives from one server to another to continue operationsto another to continue operations

Point applications to offsite server to take up rolePoint applications to offsite server to take up role Hot swap RAID drives in case of single drive Hot swap RAID drives in case of single drive

failurefailure Hot standby RAID drive to take up activity fasterHot standby RAID drive to take up activity faster Fault Tolerant Server for LAN, HDD, PCI, CPU, Fault Tolerant Server for LAN, HDD, PCI, CPU,

memory, SCSI redundancy memory, SCSI redundancy

Deployment RisksDeployment Risks

Team skillsTeam skills Time available and possible Time available and possible

distractionsdistractions Money versus performanceMoney versus performance Delivery of equipmentDelivery of equipment Applications fail in new environmentApplications fail in new environment Network traffic changes which cause Network traffic changes which cause

solution degradationsolution degradation

Staff OptionsStaff Options

NEC Professional NEC Professional ServicesServices

EfficientEfficient FastFast Easier on team Easier on team

resourcesresources

In house TeamIn house Team Know solution fine Know solution fine

detailsdetails Learning Learning

ExperienceExperience Ongoing Ongoing

Maintenance Maintenance benefitsbenefits

Quicker to refine Quicker to refine requirementsrequirements

Team SkillsTeam Skills Electrical Engineering Electrical Engineering

skillsskills Microsoft Operating Microsoft Operating

System skillsSystem skills Software / Programming Software / Programming

skillsskills Genesys skillsGenesys skills Cabling abilitiesCabling abilities Testing abilityTesting ability Fault findingFault finding Project ManagementProject Management

Time ManagementTime Management

Allocate staff to handle deployment with Allocate staff to handle deployment with some spare time for contingenciessome spare time for contingencies

Allocate resources to take care of teams Allocate resources to take care of teams normal dutiesnormal duties

Raise any project plan issues earlyRaise any project plan issues early Weekly meeting to confirm project Weekly meeting to confirm project

progressprogress Central Document Register with Central Document Register with

Versions and ownersVersions and owners

Project Implementation Project Implementation TestingTesting

1 server was used for Standard Operating 1 server was used for Standard Operating Environment testingEnvironment testing

3 servers were used in Development 3 servers were used in Development testingtesting

3 servers were used as the first site Pilot3 servers were used as the first site Pilot Pilot ran for 1 month and full rollout signed Pilot ran for 1 month and full rollout signed

offoff Network monitoring was performed before Network monitoring was performed before

and during the pilot phaseand during the pilot phase Implementation documentation was refinedImplementation documentation was refined

LogisticsLogistics

NEC server division having knowledge NEC server division having knowledge of server deployment scheduleof server deployment schedule

Exact server configuration known Exact server configuration known including drive placement, Memory including drive placement, Memory sticks and slots, RAID setupsticks and slots, RAID setup

NEC Server division can assemble all NEC Server division can assemble all options, create RAID and test servers options, create RAID and test servers before deliverybefore delivery

This saves over 1 hour per serverThis saves over 1 hour per server

Implementation Implementation DocumentationDocumentation

Large Word document with linked Large Word document with linked Spreadsheet for all site parametersSpreadsheet for all site parameters

Includes initial preparation, equipment Includes initial preparation, equipment inspection, racking, Standard Operating inspection, racking, Standard Operating Environment, Standard Application Environment, Standard Application Environment, testing, milestones and Environment, testing, milestones and expected completion time for each stageexpected completion time for each stage

Each item checked off when completedEach item checked off when completed Milestones broadcast to project team by Milestones broadcast to project team by

SMSSMS

Site DocumentationSite Documentation

Technology MapTechnology Map Hardware Hardware

SpecificationSpecification Software Software

SpecificationSpecification System MonitoringSystem Monitoring System ProcessesSystem Processes Regular Regular

OperationsOperations

New Server New Server ConfigurationConfiguration

Diagnostic Diagnostic OperationsOperations

Disaster Recovery Disaster Recovery Plans Plans

Document Document ReferencesReferences

ContactsContacts

Initial Site PreparationInitial Site Preparation Confirm all hardware is present on siteConfirm all hardware is present on site Setup and test KVM switch, monitor, Setup and test KVM switch, monitor,

keyboard and mousekeyboard and mouse Label all cables, patch panel Label all cables, patch panel

connections and hardwareconnections and hardware Setup server Configurations disksSetup server Configurations disks Confirm operation of setup Virtual Confirm operation of setup Virtual

MachinesMachines Confirm security arrangements for Confirm security arrangements for

extended hours of operationextended hours of operation

Speed ImprovementsSpeed Improvements

Try to acquire no activation Operating Try to acquire no activation Operating SystemSystem

Run deployments in parallelRun deployments in parallel Allocate other task suggestions while Allocate other task suggestions while

waiting for software to install such as waiting for software to install such as cablingcabling

Have breaks during longer install sectionsHave breaks during longer install sections Use a network switch between the Virtual Use a network switch between the Virtual

Machine and the servers to set up Machine and the servers to set up concurrentlyconcurrently

Software Deployment Software Deployment StyleStyle

There were 5 server types to setupThere were 5 server types to setup All server had the same SOEAll server had the same SOE Software installed depending on role and siteSoftware installed depending on role and site Script used to prompt for site name and roleScript used to prompt for site name and role Script the used silent command options to Script the used silent command options to

install applicationsinstall applications Script finished with text file noting each Script finished with text file noting each

application as being installed successfully or application as being installed successfully or notnot

Manual application installation method also Manual application installation method also notednoted

Testing PointsTesting Points Confirm RAID presentConfirm RAID present Confirm Operating System Installed Confirm Operating System Installed

basicbasic Confirm WSUS deploys updatesConfirm WSUS deploys updates Confirm Anti Virus presentConfirm Anti Virus present Confirm ESMPRO says hardware is OKConfirm ESMPRO says hardware is OK Confirm Server Check.vbs says Confirm Server Check.vbs says

Operating System is OKOperating System is OK Confirm applications all successfully Confirm applications all successfully

installedinstalled

Connection to LAN WANConnection to LAN WAN Test network connections to and from serverTest network connections to and from server Test main applications end to endTest main applications end to end Test support utilities with sample filesTest support utilities with sample files Initial connection to system tested achieved Initial connection to system tested achieved

in less that 1 hourin less that 1 hour Allow for client testing when you as satisfied Allow for client testing when you as satisfied

Useful designed toolsUseful designed tools Find Latest File – Checks a list of Find Latest File – Checks a list of

directories for the newest file to confirm directories for the newest file to confirm logs and backups are being created. logs and backups are being created. Results are in a colour coded relative to Results are in a colour coded relative to age HTML page.age HTML page.

Search Logs – Can scan log files for Search Logs – Can scan log files for errors or confirm connection activityerrors or confirm connection activity

Nightly zipped logs – Conserves drive Nightly zipped logs – Conserves drive spacespace

Delete Old Logs – Nightly remove zips Delete Old Logs – Nightly remove zips older than x daysolder than x days

UtilitiesUtilities

AutoIt – Scripting to imitate keyboard and AutoIt – Scripting to imitate keyboard and mouse activity to perform regular tasks. It mouse activity to perform regular tasks. It can work by itself or as an ActiveX with can work by itself or as an ActiveX with another language.another language.

Word ViewerWord Viewer Excel ViewerExcel Viewer PSTools including PspasswdPSTools including Pspasswd VNCVNC WinzipWinzip Adobe Acrobat ReaderAdobe Acrobat Reader

Post Implementation Post Implementation ReviewReview

Little things can slow progress down at sitesLittle things can slow progress down at sites Implementation plan is a good stating point Implementation plan is a good stating point

but advanced skills are needed for but advanced skills are needed for contingenciescontingencies

Personal preferences in desktop presentation Personal preferences in desktop presentation should not interfere with the implementationshould not interfere with the implementation

Some testing may identify new issues outside Some testing may identify new issues outside of server scope such as WAN latencyof server scope such as WAN latency

New servers take a short period of time to be New servers take a short period of time to be acknowledged on the WANacknowledged on the WAN

The FutureThe Future

Call centre voice recording on a very large Call centre voice recording on a very large scalescale

At least 16 servers, interfacing equipment At least 16 servers, interfacing equipment and a large Storage Area Networkand a large Storage Area Network

Central collection overnightCentral collection overnight Attached data with audio filesAttached data with audio files Search for call audio by extension, caller, Search for call audio by extension, caller,

datedate Very useful for compliance / quality Very useful for compliance / quality

purposespurposes

Storage Array Network NEC Storage Array Network NEC S2400S2400

4 to 120 Hard drives4 to 120 Hard drives 27 Terabytes 27 Terabytes

maximummaximum 8 GB RAM maximum8 GB RAM maximum 4 x 2Gbps fibre 4 x 2Gbps fibre

channels connections channels connections Supports RAID 0, 1, Supports RAID 0, 1,

5, 6, 105, 6, 10 RAID 6 has 2 parities RAID 6 has 2 parities

which allows 2 drives which allows 2 drives to failto fail