NCSA Parsons 090905
Transcript of NCSA Parsons 090905
-
8/2/2019 NCSA Parsons 090905
1/38
Dr Mark Parsons
Commercial Director, EPCC
[email protected]+44 131 650 5022
The mechanics
of EPCC
How EPCC learnt to do technology transfer
and software engineering the hard way
-
8/2/2019 NCSA Parsons 090905
2/38
The mechanics of EPCC 2
Structure of talk
What is EPCC today
Learning to deliver on time
The structure of a commercial project
Software development and OGSA-DAI
Project management
Questions and discussion
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
3/38
The mechanics of EPCC 3
EPCC Activities
Europes largest, most successfulsupercomputing centre 15 years old
Vital statistics: 65 staff 3.2M turnover (almost) all from external sources
Multidisciplinary and multi-funded with a large spectrum of activities and a critical mass of expertise
Strong engagement with industry
from local SMEs to large multinationals project based consultancy services
Supports research at University of Edinburgh viaaccess to facilities training and support TRACS visitor programme
NeSC founding partner of National e-Science Centre
Wide variety of leading-edge systems 1,600 processor HPCx system 2,000 processor IBM Bluegene/L 12,000 processor QCDOC
New investment in Advanced Computing Facility
EPCC has a unique breadth of expertise in highperformance computing
TechnologyTransfer
Training
EuropeanCoordination
HPCResearchFacilities
VisitorProgramme
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
4/38
The mechanics of EPCC 4
Commercial activities today
Bespoke software development and software project management for
business network, cluster and high-performance computing
novel application areas
from mushrooms to internet packets
Start-to-finish projects
full software development lifecycle, 3 - 12+ months most commercial projects are < 6 months
Operate like a business Commercial Group brings in business, Software Development Group delivers
charge at commercial rates ($1,000 per day)
very delivery focused
all commercial contracts are fixed cost
funded by cash contracts, public funds and European Commission, EU
many of the smaller projects are supported by SE
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
5/38
The mechanics of EPCC 5
Clients
USA:
o Cisco Systems Inco Sun Microsystems Inco IBM Corporationo Oracle Corporation
o Hewlett Packardo Microsofto Xilinx Corporation
Japan:
o Hitachi
o NEC Europeo Fujitsu Labs EuropeEurope:
o European Commission
+ many EU project partners
UK:
o Almond Engineering Ltd
o Altamira Ltdo Arran Aromatics Ltd
o Callanders Sawmills Ltd
o Calman Ltd
o CB Technology Ltd
o Centre for Customer Awareness Ltd
o CERNo Cheltenham & Gloucester plco DTIo Digital Bridges Ltdo Elektrobit Ltd
o First Group plco Golden Crumb Ltd
o High Speed Productions Ltd
o Integriti Solutions Ltd
o IP Technology Ltd
o Ironside Farrar Ltd
o Jardine Technology Ltd
o Peppers Ghost Productions Ltdo Radar World Ltdo Red Lemon Ltd
o Rosti (Scotland) Ltdo Quadstone Ltd
o SCI Ltd
o Scottish Enterpriseo The Crown Officeo TSB Bank Scotland Ltdo UK Meteorological Officeo Upstream Systems Ltd
o Alpha Data Parallel Systems Ltd
o Nallatech Ltd
2000 - 2004
UK:
o Almond Engineering Ltd
o Altamira Ltdo Arran Aromatics Ltdo Callanders Sawmills Ltdo Calman Ltdo CB Technology Ltdo Centre for Customer Awareness Ltdo CERNo Cheltenham & Gloucester plco DTIo Digital Bridges Ltdo Elektrobit Ltd
o First Group plco Golden Crumb Ltdo High Speed Productions Ltdo Integriti Solutions Ltdo IP Technology Ltdo Ironside Farrar Ltdo Jardine Technology Ltdo Peppers Ghost Productions Ltdo Radar World Ltdo Red Lemon Ltd
o Rosti (Scotland) Ltdo Quadstone Ltdo SCI Ltdo Scottish Enterpriseo The Crown Officeo TSB Bank Scotland Ltdo UK Meteorological Officeo Upstream Systems Ltdo Alpha Data Parallel Systems Ltdo Nallatech Ltd
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
6/38
The mechanics of EPCC 6
Business Strategy
to solve business problems NOT sell technology individual solutions for clients
Technology Push Down
OGSA-DAI
IPO
First Group
PGPGridSunDCG
MicrosoftC&GAutoscreen
CCA X,000
X00,000
X,000,000
AcademicResearch
Project size
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
7/38The mechanics of EPCC 7
How do we work?
No.
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
8/38The mechanics of EPCC 8
How do we work?
Take pride in a professional approach Work in small project teams
Project leader, 1-6 developers, technical reviewers
Use documented engineering & management processes Project management based on PRINCE2
Engineering using agile methods
Built from experience and industry best practice Iterative/staged development techniques
Requirements triage Test-driven development
Tuned to the leading edge of innovative software development
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
9/38The mechanics of EPCC 9
Who does the work?
Currently around 4 business development staff 2 focus on business development
2 focus on marketing and publicity
Currently around 20 engineering staff
Three full-time project managers, two software architects c. 15 consultants and principal consultants
Staff backgrounds maths , physics, computer & life sciences
Over 100 staff-years of experience, over 1/3 from industry
Typical skills Java, C/C++, Visual Basic/C#, Perl, Fortran
Distributed computing, webservices, XML, J2EE, MPI, OpenMP
Databases, SQL, JDBC, XML-DB
Software engineering, OO design, UML
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
10/38The mechanics of EPCC 10
EPCCs early history
Established in 1990 focus for interest in parallel computing within Physics and CS
Early years largely supported by UK Government Parallel
Applications Programme
made lots of money working with large UK corporations tooptimise/parallelise their codes
How did our funding model come about? from a belief in the self-funding of University
research
weve shown it can be done but its verydifficult
it did mean we had to work with industry fromthe beginning
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
11/38The mechanics of EPCC 11
EPCC history (continued)
1990-1994 funded by UK Government Parallel Applications Programme
grew to 65 staff
many parallelisation projects with UK industry aerospace, nuclear,oil & gas etc etc
span out company Quadstone 1995-1996
as Gov money dried up so did projects
had to move from long term projects (18 months) to much shorter
projects (3-6 months) major problem project / cost overruns
nearly had to make many staff redundant
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
12/38The mechanics of EPCC 12
EPCC history (continued)
1997-2000 successfully moved markets from large-scale industry to SMEs
opportunities focussed around successful EU TTN project
projects 3-6 months in duration
embarked on having a repeatable process
2000-now over the past few years moved into Grid computing
continued to work with industry
wide variety of projects:
OGSA-DAI data access & integration for the Grid Intersim packet level modelling of differentiated services
Golden Crumb automatic mushroom selection in factory
Cheltenham & Gloucester data mining for mortgage industry
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
13/38The mechanics of EPCC 13
How does EPCC work today?
We have well developed project processes
Two linked processes software development process
project management process
Will illustrate software development process using OGSA-DAI as example
Recently moved to PRINCE2 project management
methodology
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
14/38The mechanics of EPCC 14
The project lifecyle
Commercial Group identifies clients and initiates discussions
Following initial discussions CD and technical staff visit
company to discuss requirements
High level design written timings / costs agreed
may involve free code survey at this point Contract negotiated fixed price includes detailed
workplan based on design
Project handed to IS staff scheduled according to skills
All projects have Project Leader, Applications Consultant, Technical Reviewer
Regular meetings between IS and CG
CG act as account manager to company / funder
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
15/38The mechanics of EPCC 15
OGSA-DAI
Data Access and Integration for databases resources on the Grid
Aim to deliver application mechanisms that: Meet the data requirements of Grid applications
Functionally, performance and reliability
Reduce development cost of data centric Grid applications
Provide consistent interfaces to data resources
Acceptable and supportable by database providers
Trustable, imposed demand is acceptable, etc.
Provide a standard framework that satisfies standard requirements
A base for developing higher-level services
Data federation / Distributed query processing Data mining
Data visualisation
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
16/38The mechanics of EPCC 16
OGSA-DAI team
IBM Development Team, Hursley
NEReSC, Newcastle
NeSC, EdinburghEPCC Team, Edinburgh
ESNW, Manchester
IBM Dissemination Team
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
17/38The mechanics of EPCC 17
Software Process and Teams
Testing
Reqs.
Prototype
Prioritisation
Fix Bugs
UseCases Requests
Design Implement QA
Release
SupportTest Cases
Programme
Board
Technical
Review Board
Technical
Reviewer
DEVELOPERS
USERS
REV
IEW
Contribs
Ingest
Dissem.
Training
Nightly unit +system tests
Additionaltest cases
System testsbased on reqs
Continual process
Deep trackfeatures
Users
Group
Peer Review
and Inspection
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
18/38
http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
19/38The mechanics of EPCC 19
An agile approach to development
Agility is all
Grid/HPC environments and problems = complex systems
complex systems = big, complex projects
big, complex projects = high risk of failure
adopting incremental approaches to requirements, design, and
implementation helps minimise risk delivering small increments regularly is good
good for quality, for visibility, for morale
Keep your eyes on the road
keep an active eye on project risks think about what happens if this goes wrong
just thinking about it reduces the likelihood itll happen!
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
20/38The mechanics of EPCC 20
Releasing software
No release schedule = no releases
dont timebox research, but do timebox development HPC is fun and exciting - beware feature creep!
hows the project?
oh, were 95% there (and always will be)
frequent release milestones focus developers
but dont overspecify what will be released
OGSA-DAI had the opposite problem three months too short
six months about right major/minor/patch/special brew
set your testing timetable in stone
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
21/38The mechanics of EPCC 21
Requirements, requirements, requirements
write em down! Give em numbers! remember, requirements arent just functional!
whatever they are, they are always testable
tests on HPC systems may be tricky, but thatmakes it fun!
MoSCoW notation is good Must, Should, Could, Won't
how important are Priority 3 requirements again..?
OGSA-DAI had lots of requirements
but make sure you can understand their worth real users are often better than good ideas
a user group helps to focus development as software matures
Know your requirements
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
22/38The mechanics of EPCC 22
Return, recycle, reuse
Throwaway prototypes never are
once Ive proved this, Ill junk the code
no, you wont (or your grad student wont)
apply some basic process even to trivial codes
even reuse of good code is sometimes wrong
OGSA-DAI started with high ideals beware the big ball of mud
patterns in architecture
Shantytown
enables quick exploration of feature territory must be built on a strong central foundation
must include council legislation aka testing
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
23/38The mechanics of EPCC 23
OGSA-DAI Dashboard
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
24/38The mechanics of EPCC 24
Can I see your documents please?
Document! Document! Document!
Imagine trying to program without a language reference
structure and stability is good
Get people who like writing documents to do them
but get everyone to doc their code
a single editor can provide guidance Good code documentation can be used by the tooling
Good human documentation will win your users support
Make sure you dont underestimate the cost code maintenance and documentation takes longer than code
development
make it part of the process
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
25/38The mechanics of EPCC 25
People power
Social engineering is the key
Push decisions down to the developers
Too many chiefs
make sure you know what are the key battles to win
Have a process for change
or one person will become very unpopular developers and managers both think they know whats best
Understand your teams
different people like working in different ways
no one style for management in OGSA-DAI
Competition is good
go one better
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
26/38
The mechanics of EPCC 26
The big picture
Balance the hype
software engineering is about vision vs effort vs requests expectation management is important
researchers, developers, users and funders are all different
and all want different things
the larger the project, the harder it falls
Listen to your users useability is good
it has to install easily
dont change your interface
client tooling helps
support helpdesk is better
user groups are interesting
S f d l
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
27/38
The mechanics of EPCC 27
Software development summary
Agile methods are very sympathetic
the Agile founders disliked Rigid Inflexible Processes too!
Adopt a simple process and toolset even lightweight process really pays off
scoping, requirements and risk analysis up front
incremental approach to design, develop, test
learn some basic tools (theyre even free!)
distributed teams are hard to manage strictly
distributed management is even harder
Listen to your customers they always know best
P j M
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
28/38
The mechanics of EPCC 28
Project Management
All technical staff have a line manager and at least one
project leader
Procedures are well documented and have grown up over
time
Recently we have moved to PRINCE2 project managementmethodology for commercial projects
seems to work well but is a bit of a culture shock
We employ staff specifically for project management
All staff time is logged planned and actual
A working day has two blocks of 3 hours
Staff can bid for time to do research / proposal writing
Wh i PRINCE2?
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
29/38
The mechanics of EPCC 29
What is PRINCE2?
PRojects IN Controlled Environments version 2
A project management standard produced by UKs Office ofGovernment Commerce (part of DTI)
PRINCE2 is a process-based approach for project managementproviding an easily tailored, and scalable method for the managementof all types of projects
PRINCE2 is a de factoUK PM standard becoming mandatory in the public sector (Gov, NHS, Police)
becoming PM method of choice in business
Unilever, GlaxoWellcome, Tesco, BT, Sun, TSB, NatWest,
Norwich Union, Centrica, Cable & Wireless becoming widespread in Europe too
PRINCE2 is internationally recognised and respected
Wh t i PRINCE2 t?
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
30/38
The mechanics of EPCC 30
What is PRINCE2 not?
PRINCE2 is not a software engineering method
but it grew out of an IT environment
and it fits well with traditional or agile development methods alike
PRINCE2 will not help you code better but it will help you deliver better quality products, on time
and will stop you falling out with your boss/staff
PRINCE2 will not tell you how to write software but it will leave you alone to write software your way
PRINCE2 is not a silver bullet but its general, flexible and tailorable and most importantlyits
based on common sense
PRINCE2 i t h ll
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
31/38
The mechanics of EPCC 31
PRINCE2 in a nutshell
Projects have a clear Business Case or they dont happen
remind me again why were doing this project?
Projects have a beginning, a middle and an end clearly defined they start and they stopthey dont weeble on forever
Projects run in stages with clearly defined boundaries get a clear picture of how were all doing
Product-based planning focuses on deliverables not tasks think what do we have to make?
Layered management: corporate, board, project, team each level has a clearly defined interface with the others
Management by exception if there are no problems, just carry onmanagement dont meddle
Change is fundamental: change management is intrinsic assume things will change and plan accordingly
Th PRINCE2 di
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
32/38
The mechanics of EPCC 32
The PRINCE2 process diagram
Corporate or Programme Management
Directing a Project
Starting upa Project
Planning
Initiating aProject
Controllinga Stage
Managing
Product
Delivery
ManagingStage
Boundaries
Closing aProject
Project
Mandate
PRINCE2 C t
http://www.epcc.ed.ac.uk/http://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.prince2.com/whatisp2.htmlhttp://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
33/38
The mechanics of EPCC 33
PRINCE2 Components
As well as the processesthere are several complementary
components The Business Case
a key driver the Why?for the project
either a genuine (commercial) business case or at least a set ofcompelling reasons
owned by the Executive
monitored throughout the project
if the BC goes away, the project should be stopped
The Project Organisation describes the four management layers
corporate, board, project, team
everyone should have a job description
make roles and responsibilities clear
PRINCE2 C t (2)
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
34/38
The mechanics of EPCC 34
PRINCE2 Components (2)
Plans
product-based, as discussed above
write product descriptions for key products
Controls divide the project into Management Stages
a Stage is as far ahead as you can plan in reasonable detail
typically a few months
define reports, meetings etc.
Tolerances allowed variations in time, budget, scope before escalation triggered
you have six months, +/- 1 month
you must satisfy these requirements; those are optional this stage
PRINCE2 Components (3)
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
35/38
The mechanics of EPCC 35
PRINCE2 Components (3)
Quality
the project must define methods for QC and test
quality checks should be built in to the MP process
Risk think about it, monitor it
one of the best management tools is to ask what might go wrong?
and create plans to handle it if it does
Configuration Management keep track of product versions and histories
software version control tools are a good way of implementing this
PRINCE2 Summary
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
36/38
The mechanics of EPCC 36
PRINCE2 Summary
PRINCE2 is a powerful, flexible, scalable PM approach
Its based on industry best practice rooted in software development projects
Provides good, intelligent layers of management control
Formalises, in a positive way, customer relations
Can fit easily with agile software development
Its the only PM approach with internationally recognised
qualifications
Final comments on working with industry
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
37/38
The mechanics of EPCC 37
Final comments on working with industry
Wear a tie!
Remember that the person youre meeting is just as nervous
of meeting a mad academic as you are of meet a rapacious
capitalist
The managing director of the company may be drunk
Always apply Denis Healeys law of holes: When in one stop
digging
If youre going to deliver late tell the customer straightaway
Listen listen listen!!! Its the only way to get business
Questions / discussion
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/ -
8/2/2019 NCSA Parsons 090905
38/38
Questions / discussion
?
http://www.epcc.ed.ac.uk/http://www.epcc.ed.ac.uk/