How HPC Transforms the Corporate Information Technology Ecosystem
-
Upload
inside-bigdatacom -
Category
Technology
-
view
770 -
download
0
Transcript of How HPC Transforms the Corporate Information Technology Ecosystem
Imagination at work
HPC in EnterpriseHow HPC transforms the Corporate Information Technology EcosystemThomas Leung – Sept 10, 2015
make happen
© 2015 General Electric Company - All rights reserved
Who am I?
'No one ever changed the world without first being labeled a 'troublemaker‘ ~ Author
Unknown
© 2015 General Electric Company - All rights reserved
Agenda
What is Industrial HPC
Technology, design and
architecture
Position of HPC at enterprise
HPC as a service
Challenges and future focus
*IDC HPC and ROI Pilot Study 10/2013
© 2015 General Electric Company - All rights reserved
Average of
revenue per
HPC dollar is
$462.4*
What is industrial HPC and difference to Research HPC?
© 2015 General Electric Company - All rights reserved
Industrial HPC (aka Commercial / Enterprise HPC)
• Commodity hardware
• Small – large systems
• Co-located data center
• Resource management based on business priorities between business units. (long
run time.)
• Computational process is part of the design workflow.
• High availability and stability with 24 /7 operation. Business critical processes /
projects support
• Multiple years planning with yearly funding / commitment. (in most cases)
• Multiple source of funding with multiple plans/ directions.
• Highly integrated with enterprise IT systems / internal infrastructure.
• Legacy code and latest code support
© 2015 General Electric Company - All rights reserved
System is busy all the time….
Industrial HPC systems operates 24 x 7 with over 95% utilization.
© 2015 General Electric Company - All rights reserved
HPC expansion & requirements
HPC systems grow all the time with requirements of both types of HPC.
Research HPC
requirements
Enterprise IT
Requirements
Business Requirements
Industrial HPC
requirements
© 2015 General Electric Company - All rights reserved
Bleeding edge technology focus on scale and speed. HPC system will see more “defects” in future.
ResearchHPC
© 2015 General Electric Company - All rights reserved
Pull and Push force – on design decision
New technology adoption and wide range of applications support made HPC system design harder.
Vendors: I have the
latest and greatest
technology. Trust me!
It works.
Users: Why aren’t
we using the latest
greatest
technology?
Cloud provider:
Cloud can do
everything now. Finance:
Faster,
More,
Cheaper are
always
better!
Business Leaders:
It is stable now, I
don’t want
anything new and
risky.
Engineer: I have
to support codes
developed 20
years ago.
Managers:
Downtime is not
acceptable.
© 2015 General Electric Company - All rights reserved
HPC changes enterprise cyber security
More collaborators, high bandwidth, low latency connections leads to new challenges.
HPC systems
Enterprise
Defense
Global Users
Machines
National Labs
Enterprise
Network
Enterpris
e Network
© 2015 General Electric Company - All rights reserved
HPC systems
HPC is moving Enterprise IT
Networking Storage Infrastructure
/Cloud
Application
Mostly 10Gb NAS , Iscsi, FC
storage
Co-located data center
with low power
requirement
Java, .net
10Gb / 40Gb
Ethernet only
Everything is
backup
Virtual machine Max 244GB ram
(EC2)
1 - 10Gb internet
connection
TB – PB storage VM hypervisor controls
resource
No MPI
programming, Java
IaaS only IaaS only PaaS only SaaS only
HPC
40Gb / 100Gb
Ethernet with FDR /
ERR Infiniband
Some workflow
need TB ram
PB storage with fast
IO
Fortran,
Multi thread with
MPI Programming
Resource Mgr
controls
PaaS / SaaS
Enterprise IT
© 2015 General Electric Company - All rights reserved
Marriage between Big Data and HPC……
Distributed storage and processing framework in a stable & scalable environment.
Big Data system makes “data” more accessible to HPC. It enhances the simulation efficiency and effectiveness.
Big DataHPC
© 2015 General Electric Company - All rights reserved
Position of HPC in Enterprise
ERP / Enterprise
systems
HPC
inside
Past: ERP / Enterprise systems are
the center of the business workflow /
decisions.
HPC is just one of the engineering
tool kits.
Future: Enterprise Hadoop platform
becomes the “CENTER” of all decisions
and workflow.
© 2015 General Electric Company - All rights reserved
Understand availability and redundancy of HPC
Single Cray system outage
Recover 100TB data from Tape
Fixing incorrect job submission
Fixing Job submission errors
HPC issues associate with system down
time
Not a lot of system issues can be fixed in less than 1
min.
This is the area of improvement!
© 2015 General Electric Company - All rights reserved
HPC team evolving
HPCers are hard to get. Talent search for HPC positions are harder.
system administrators, storage administrator, database administrators, software
developers, network administrators, solution architect, technologist, business
analyst, project managers, researcher, computer scientist, leaders.Modern HPC
team
HPC team is usually a group of
geeks…..
Traditional
HPC team
Future key requirements:
• Big Data knowledge & hands on experience – able to build production hadoop system and support
operation
• Programming – manage systems with provided API
• Application support – Optimization, New platform adoption and migration
• Data science expertise, business expertise
• Experience with large systems – Really large systems
• Industry specific domain knowledge – Enhance partnership with users and business.
© 2015 General Electric Company - All rights reserved
Focus areas for industrial HPC
High availability
• Establish automated
failover capability
• Enhance Job level
recovery after system
failure
• Don’t forget all
surrounding IT services.
24 x 7 service
• 24 x 7 system
monitoring
• 24 x 7 user support
• Full debug and
compiling environment
available
Multiple systems strategy vs
single system strategy
• Easy for compute resource
but hard for storage
solutions.
• Multiple location planning
Automation
• HPC service shall be
automated as Web
service
• API driven management
activity / functions will
increase
© 2015 General Electric Company - All rights reserved
Reduce cost
• Software License
cost
• Storage cost
• Platform cost
• $ per CPU cores
Call for actions
Resource Manager
Big Data
VDI
HPCCloud
Storage
© 2015 General Electric Company - All rights reserved
Research HPC
• Commodity hardware
• Huge Systems
• Dedicated Data Center
• Resource management based on fair share between users /
programs. (relatively short run time.)
• Handle computational process / workflow only.
• Extended MTBF with parts fail all the time.
• Multiple years planning with one-time funding / commitment.
• Centralized source of funding – University / Government.
• Independent from IT systems / infrastructure.
• latest code support
© 2015 General Electric Company - All rights reserved
Size of HPC systems
Small
~100 cores
Eg: Workgroup
<$100k
Medium
~500 cores
Eg:
Departmental
$100k - $250k
Large
~1000 cores
Eg:
Divisional
$250k - $500k
X- Large
>2000 cores
Eg:
Supercomputer
>$500k
XX – Large
>10000 cores
Commercial HPC
Research HPC
Research HPC covers National Lab, Government Agency
and universities
Industrial, Finance, Oil &Gas, Manufacturing companies
© 2015 General Electric Company - All rights reserved
Image citation / source CDN Locations | EdgeCast, www.edgecast.com
Gauging Skills | Autism After 16www.autismafter16.com
Service Availability: I'll See Your Five 9s and Raise You One More ...aits.org
MySQL Master/Slave Management with mysql-master-ha | CRGNetwww.cirgan.net
Illustration Depicting Two Triangular Warning Road Signs With A ...www.shutterstock.com
Review: Thomas Pynchon's “Bleeding Edge” | The American Readertheamericanreader.com
Can Crowd Testing replace an Internal Quality Team? | 99testsblog.99tests.com
VIDEO: Find the balance between push and pull can help your ...www.teamsupport.com
FCoE Adoption Curve - Wikibonwikibon.org
Online Customers are Like Toddlers, They Don't Like Getting ...www.contenthook.com
Stkit MOC] Iron Man Hulkbuster statue: A LEGO® creation by St Kit ...www.mocpages.com
LEGO Ideas - Iron Man Hulkbuster armorideas.lego.com
How does an ERP system work? - Quorawww.quora.com
hadoops-role-in-enterprise-architecture-7-638.jpg?cb=1364477105www.slideshare.net
Get Free or Cheap Legal Fees Help in the USwww.grantfundingexpert.org
Migration | QualITasqualitas.co.id
Respect for system administratorsgeekshumor.com
Sincerely Stephanie: The Digit-al Dozen September Spectacular: Geekswww.sincerelystephanienails.com
Huntington's Disease Youth Organization - Young adults - Ask a ...en.hdyo.org
ABigFave | Flickr - Photo Sharing!www.flickr.com
Peak Hour Protests Immobilise Bangalore - The New Indian Expresswww.newindianexpress.com
Competitive Cost Benchmarking Software for Manufacturerswww.mtisystems.com
Zenith Software announces India Expansion PlansDATAQUESTwww.dqindia.com
© 2015 General Electric Company - All rights reserved